Ideal transfer of call handling from automated systems to human operators based on forecasts of automation efficacy and operator load

ABSTRACT

The present invention relates to dynamic policies for transferring people from an automated or user-directed call handling system to a human operator, depending on considerations of the likelihood of the failure of the interaction with the call-handling system, predictions about the expected time or frustration associated with using the system, and the current load on human operators. Systems and methods leverage probabilistic models of system and user behaviors built from logged data. A decision-theoretic analysis and corresponding models of ideal decisions about the transfer of calls from an automated system to a human operator are provided. The methods have application to a spectrum of call-handling systems including touch-tone and speech-recognition-based systems.

TECHNICAL FIELD

[0001] The present invention relates generally to systems and methodsthat facilitate communications between devices, systems, processes,and/or individuals. More particularly, the present invention relates toemployment of dynamic policies for transferring people from an automatedcall routing system to a human operator given the likelihood of success,the effort associated with continuing to interact with an automatedsystem and the wait times associated with entering a queue for a humanoperator.

BACKGROUND OF THE INVENTION

[0002] Many users have had experiences with the use of automatedspeech-recognition based call routing systems for accessing data andservices by telephone. Such systems have proliferated lately with theadvent of more accurate spoken command-and-control and continuous speechrecognition systems for limited domains. Even when the scope of anapplication is restricted, however, such systems can produce frustratingfailures. To address user frustration, many designers put in placeoptions for callers to be immediately routed to a live, human operator.Beyond speech recognition systems, other automated systems have beendeployed such as the use of touch-tone driven menu systems. Such systemsmay frustrate users in providing a fixed set of options that either maynot appear to offer the right options, offer options that haveunderstandable relevance, and/or may appear to route the caller to afrustrating sequence of options rather than converging quickly to adesired setting.

[0003] A growing number of organizations are using automatedcall-routing systems that employ touch-tone routing or speechrecognition and natural language processing to assist users withnavigation. Automated methods are employed to reduce the load onoperators or to reduce costs associated with staffing of call receptionand routing functions. However, failures and frustrations of automatedreception and routing can be costly to callers and to organizationsemploying them as callers may have to put in more effort, wait longertimes, or simply hang up in frustration and take their businesselsewhere.

[0004] There has been a recent growth in the popularity of deployingspeech recognition systems. One reason for the popularity of suchsystems is their promise at providing a more natural interface thantouch-tone routing. However, organizations have been discovering thatvoice routing still falls short of the quality of service that a live,human operator can provide. In one example, an automated call routingsystem may field all internal directory assistance calls. Using speechrecognition, the system attempts to uniquely identify one of overthousands of possible name entries in a company's global address book.Needless to say, a vocabulary of this size is bound to have deleteriouseffects on the performance of the speech engine.

SUMMARY OF THE INVENTION

[0005] The following presents a simplified summary of the invention inorder to provide a basic understanding of some aspects of the invention.This summary is not an extensive overview of the invention. It is notintended to identify key/critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentsome concepts of the invention in a simplified form as a prelude to themore detailed description that is presented later.

[0006] The present invention relates to guiding calls in accordance withan automated call routing system. One or more decision models areprovided that output policies for switching people from an ongoingautomated system to a human operator based on context-sensitive analysisof the spoken dialog situation at hand. This also relates to applyingdecision-theoretic principles to reason about the ideal melding ofpeople and automated reasoning systems. Thus, callers' experiences witha call routing system are optimized by developing models that considerthe probability that callers will be ultimately successful (or not) intheir collaboration with the automated system, the expected number ofsteps required for that success, and the current load on humanpersonnel. In one aspect, a voice routing system for directoryassistance can be analyzed via data logs of a system's performance. Oneor more probabilistic models can then be constructed from the data logsregarding the likelihood of success or failure with the automatedsystem, and the number of steps, overall duration, effort, orfrustration with the use of the automated system, given that there willbe eventual success of the interaction. The present invention couplespath-dependent probabilities from learned models with a decisionanalysis to optimally guide the transfer of calls to a human operator,given information about the expected current wait times associated witha transfer to a human operator.

[0007] To the accomplishment of the foregoing and related ends, certainillustrative aspects of the invention are described herein in connectionwith the following description and the annexed drawings. These aspectsare indicative of various ways in which the invention may be practiced,all of which are intended to be covered by the present invention. Otheradvantages and novel features of the invention may become apparent fromthe following detailed description of the invention when considered inconjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008]FIG. 1 is a schematic block diagram illustrating an automated callrouting and decision system in accordance with an aspect of the presentinvention.

[0009]FIG. 2 is a diagram illustrating call routing statistical data inaccordance with an aspect of the present invention.

[0010]FIG. 3 is a diagram illustrating dialog features in accordancewith an aspect of the present invention.

[0011]FIG. 4 is diagram illustrating a probability tree in accordancewith an aspect of the present invention.

[0012]FIG. 5 is a diagram illustrating a dependency network for dialogfeatures in accordance with an aspect of the present invention.

[0013]FIG. 6 is a diagram illustrating a Markov Dependency network inaccordance with an aspect of the present invention.

[0014]FIG. 7 is a prior distribution of an expected number of dialogsteps in accordance with an aspect of the present invention.

[0015]FIG. 8 is a flow diagram illustrating systematic automated callrouting according to an aspect of the present invention.

[0016]FIG. 9 is a schematic block diagram illustrating a suitableoperating environment in accordance with an aspect of the presentinvention.

[0017]FIG. 10 is a schematic block diagram of a sample-computingenvironment with which the present invention can interact.

DETAILED DESCRIPTION OF THE INVENTION

[0018] The subject invention employs decision theory in connection withmanaging calls. More particularly, management of call traffic andhandling of specific calls is important to many businesses, and thesubject invention employs decision theory in connection with optimizinghandling of calls. An automatic call answering system can be employeduntil the cost associated therewith (e.g., customer hanging up)outweighs the benefits (e.g., minimizing human intervention to deal withthe call). Thus, a system in accordance with the invention employs adecision-theoretic based framework to take a most appropriate action(e.g., use automatic call answering system, end use of call answeringsystem, switch to human operator, route call to different service . . .) based upon utilities associated with a particular component foranswering a call and/or handling of call traffic, state of caller(s),state of entity receiving call(s), bandwidth, resources, and nature ofcall, for example. Thus, the subject invention optimizes utilization ofresources via employment of decision theory and one more models that aretrained from past activity logs.

[0019] As used in this application, the terms “component,” “model,”“system,” and the like are intended to refer to a computer-relatedentity, either hardware, a combination of hardware and software,software, or software in execution. For example, a component may be, butis not limited to being, a process running on a processor, a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component. One or more components mayreside within a process and/or thread of execution and a component maybe localized on one computer and/or distributed between two or morecomputers.

[0020] As used herein, the term “inference” refers generally to theprocess of reasoning about or inferring states of the system,environment, and/or user from a set of observations as captured viaevents and/or data. Inference can be employed to identify a specificcontext or action, or can generate a probability distribution overstates, for example. The inference can be probabilistic; that is, thecomputation of a probability distribution over states of interest basedon a consideration of data and events. Inference can also refer tological inferences, including deterministic techniques employed forcomposing higher-level events from a set of events and/or data. Suchinference results in the construction of new events or actions from aset of observed events and/or stored event data, whether or not theevents are correlated in close temporal proximity, and whether theevents and data come from one or several event and data sources.

[0021] Referring initially to FIG. 1, a call routing and decision system100 is illustrated in accordance with an aspect of the presentinvention. The system 100 includes an automated call routing system 110that is routinely employed for providing automated responses 120 to oneor more callers 130. These systems include processing components,switching components, electronic directories, and associated softwaresuch as speech recognition components for communicating with the callers130 and routing calls to an identified individual at 140 (e.g., voicecommand indicating individuals last name). If callers 130 have troublemaking a connection with an individual or party, the call routing system110 can connect to a human operator 144 to provide further assistance tothe callers.

[0022] In accordance with one aspect of the present invention, one ormore decision models 150 are employed with the call routing system 110to facilitate efficient operations of the system 100, provide moreefficient coupling between callers and respondents, and mitigate callerfrustration when interacting with such systems. In one aspect, thisachieved by training the decision models 150 via a data log 160 that hasrecorded data of past activities and interactions with the call routingsystem 110. Output from the decision models 150 is then employed forcall routing determinations. Such data includes statistical informationsuch as how often speakers have been found or not found, how often anoperator has been requested and so forth wherein this data is describedin more detail below with respect to FIG. 2. After the decision modelshave been trained, the call routing system 100 works in concert with themodels to facilitate call routing between callers and individuals to becontacted. As will be described in more detail below, the models 150 canbe employed in accordance with dynamic policies relating to the costsand benefits of switching a caller to the human operator, for example.

[0023] Referring now to FIG. 2, a diagram 200 is illustrates statisticaldata that is analyzed in accordance with an aspect of the presentinvention. Although various categories of data are depicted by thediagram 200, it is to be appreciated that other categories of statisticsrelating to call routing performance can be similarly analyzed (e.g.,average amount of time to connect). To assess the overall performance ofthe system described above with respect to FIG. 1, over 250 megabytes ofdata logs covering a period of roughly one year was obtained. The logcontained close to 60,000 transcriptions of individual sessions with acall routing system, capturing respective system and caller actions.Some possible outcomes for a given session are as follows:

[0024] SpeakFound: The system finds the name in the directory.

[0025] SpeakNotFound: The system infers that the person requested maynot be in the directory, and routes the call to an operator.

[0026] OperatorRequest: The user requests an operator by pressing ‘0’.

[0027] HelpRequest: The user tries to request help by pressing ‘*’ or‘#’.

[0028] HangUp: The user quits during the session.

[0029] MaxErrrors: The system reaches the maximum number of allowedmistakes, and routes the call to an operator.

[0030] NotReady: The system is temporarily out of service.

[0031] Undefined: The user tries to press a numeric key.

[0032] The diagram 200 depicts an overall breakdown of all individualsessions into their possible outcomes. Due primarily to the large listof employee names in a sample company directory, a call routing systemwithout associated models and policies achieved a dismal success rate of45%, corresponding to the percentage of SpeakFound outcomes where thesystem correctly identified the proper name without the user attemptingto request help from a human operator. The success rate jumps to 66%when session logs in which the caller did not even attempt to speak aname are removed. The situations where no interaction is attempteddisclose the unsettling fact that many callers are likely to completelybypassing interaction with the system. In fact, these “no-name” attemptscomprised 31% of the entire data, 85% of all OperatorRequest outcomesand 53% of all HangUp outcomes. In other words, roughly one out of everythree callers decide to completely avoid interacting with the systemunaided by attendant models. A longitudinal analysis of no-name casesover several months revealed that users appeared to be learning toimmediately request a human operator. In fact, the correlation between ano-name situation and an “OperatorRequest” outcome was significant atρ=0.88. This correlation went up to ρ=0.90 after the engineerresponsible for the system added the phrase “for an operator, press ‘0’”to the introductory prompt, which had the effect of instructing usershow to avoid working with the automated system. These findings wereemployed to characterize and optimize users' experiences via the modelsdescribed above to more effectively enable callers to interact with theautomated call routing system.

[0033] Turning to FIG. 3, one or more dialog features 300 areillustrated in accordance with an aspect of the present invention. Callrouting systems typically treat policies for transferring users to anoperator in an ad-hoc manner using handcrafted rules composed of variousdialog features, such as the number of questions asked so far in asession. Rather than relying on intuition to construct the rules, thepresent invention builds probabilistic models from a large database ofsession logs with the intention of discovering dialog features that werepredictive of success or failure. The dialog features employed forlearning the models fall into four broad categories:

[0034] System and user actions at 310 (e.g., asking the user to pickbetween the top two guesses, the user has pressed a key, etc.)

[0035] Session summary features at 320 (e.g., number of name attemptsdetected, overall duration, etc.)

[0036] n-best recognitions features at 330 (e.g., range of confidencescores assigned by the speech recognition subsystem, mode, greatestconsecutive score difference, count of the most frequent first/last/fullnames, etc.)

[0037] Generalized temporal features 340 (e.g., number of recurringfirst names that match in consecutive turns, etc.)

[0038] The n-best recognitions features 330 can be derived from a speechrecognizer, and the generalized temporal features 340 were included tocover trends between n-best lists. Using varying amounts of featureinformation, three classes of models (can be more or less than three)were built to estimate the likelihood that a session with the callrouting system would eventually end in success or failure.

[0039] Referring now to FIG. 4, a probability tree 400 showing thelikelihood of success given any sequence of system actions isillustrated in accordance with an aspect of the present invention.Conditioning on a sequence of actions taken by a call routing systemsystem, a likelihood of success can be determined, that is,p(SpeakFound|E), wherein observational evidence E refers to all systemactions taken so far, by counting the number of cases along the actionsequence that resulted in success over the total number of cases alongthe sequence. The probabilities can be visually displayed as a tree 400where each branch represents the system action, as shown in FIG. 4.

[0040] Table 1 below provides an expanded view of several dialogbranches extracted from the tree 400, where n refers to the number ofcallers reaching that state in the data logs. According to theprobability tree 400, the first system action [Operator Intro], whichgives a standard prompt but appends the phrase “For an operator, pleasepress ‘0’,” has only a 45% chance of succeeding. Interestingly, if thesystem subsequently asks the caller to repeat the entire name and thenasks the caller to pick among three possible guesses, the likelihood ofsuccess increases to 66%. The increased probability of success for thisinitial interaction is atypical for dialog with the automated system asa whole. Typically, the longer the sequence of actions or path along thetree, the less likely it is that success will be achieved. TABLE 1Expanded portion of several branches of the tree capturing probabilitiesof success given path. [Operator Intro] (p: 0.45, n: 51134) [OperatorIntro] >> [Repeat full name] (p: 0.45, n: 20570) [Operator Intro] >>[Repeat full name] >> [Confirm from 3] (p: 0.66, n: 156) [OperatorIntro] >> [Repeat full name] >> [Confirm from 3] >> [Repeat first name](p: 0.17, n: 24) [Operator Intro] >> [Repeat full name] >> [Confirm from3] >> [Repeat first name] >> [Repeat first name] (p: 0, n: 8)

[0041] The advantage of the marginal probability tree model 400 issimplicity. The data that is required to build the tree 400 is asequence of system actions for respective sessions and the ultimateoutcome. As we will be described in the following section, the sequenceof system actions happens to be the most predictive factor indetermining success. The drawback to this model 400 is that the longerthe sequence of actions, or path down the tree, the sparser the set ofcases for building robust predictors of the likelihood of success.

[0042]FIG. 5 illustrates a dependency network 500 for dialog features inaccordance with an aspect of the present invention. Exploiting the fourcategories of dialog features discussed previously, another aspect ofthe present invention employs a Bayesian structure learning to buildprobabilistic models for predicting session outcome. A Bayesian learningtool that performs structure search and model scoring for differentpredictive models given a data set can be employed to learn a predictivemodel. As an example, the WinMine toolkit (Chickering et al. 1997) canbe employed to build a dependency network and associated decision treeconsidering session outcome and other dialog features as inputvariables.

[0043] The top five dialog features for predicting outcome are displayedin the diagram 500 in order of dependency strength are as follows: (1)the sequence of system actions, (2) the count or number of alternates inthe n-best recognitions list, (3) the number of times the user attemptedto speak a name, (4) the largest confidence score assigned by thesystem, and (5) the number of dialog turns—defined as a question-answerpair.

[0044] The dependency model 500 resonates with findings in theprobability tree in that the dependency with the strongest link tooutcome is the sequence of system actions. In the corresponding decisiontree for the dependency network, it was found that the first decisionsplit occurs when the system either does or does not identify the propername after one confirmation attempt. If the confirmation is successful,then the likelihood of a SpeakFound session is almost certain at 99%.Otherwise, the decision tree 500 considers other dialog features topredict success. Consistent with the idea that the longer the sequenceof actions, the lower the chance of success, it was found that two ofthe next five strongest connections were related to the length of thedialog. The remaining three features were parameters output by thespeech recognizer.

[0045] Given that the likelihood of success is closely tied to thelength of the dialog, analysis was directed at how the decision rulesplit the fifth strongest dependency, the number of dialog turns. Sincethe split occurred at 2 dialog turns, the data set was divided by thenumber of turns and models were constructed for the first turn, secondturn, and greater than two turns. TABLE 2 Top five dependencies for onedialog turn, two turns, and more than two turns. Model Top Second ThirdFourth Fifth 1 turn Number Range of Skewness Number Kurtosis of scoresof scores of dialog of scores alternates turns 2 Number Redundant NumberRedundant Count of turns of dialog first of last names n-Best turnsnames alternates Lists >2 Number Number Session Sum of Number turns ofnames of dialog duration scores of heard turns alternates

[0046] Table 2 displays the top five influencing variables for therespective dependency network models. Notice that for the first turn,almost all of the top five variables relate to the n-best recognitionlist generated by the speech recognizer, including the distributioncharacteristics of the confidence scores such as skewness and kurtosis.In moving to the second turn, however, generalized temporal featurescome into play, such as the maximum number of times a first or last namefrom the first dialog turn shows up again in the second turn. After twoturns, the dialog features relate mostly to the length of the dialog.

[0047] Referring to FIG. 6, a Markov Dependency network 600 inaccordance with an aspect of the present invention. In order to fleshout features from the speech recognizer that might be predictive ofperformance in the next turn, temporal dependency networks for Markovpairs of n-best recognition list features were constructed. FIG. 6displays the Markov dependency network 600. Since the count or number ofalternates in the n-best recognition list was consistently selected tobe one of the most strongest dependencies, the dependency network showshow it is possible to predict the number of alternates in turn t fromn-best list features in turn t−1.

[0048] In order to evaluate the predictive performance of theprobabilistic models described above, the classification accuracy of thelearned models was compared against a marginal model capturing theoverall run-time statistics for the training set. The models resultedfrom splitting the original dataset 70/30 into training and holdout datasets. As the probabilistic models were targeted for guiding decisionsabout dispatch, the outcome variable was circumscribed into two possibleclassification states: success versus failure, where success correspondsto the SpeakFound state. Table 3 below presents the results of theclassification task on the holdout data for models examining the firstdialog turn, the second turn, greater than two turns, and finally, thecomplete dataset.

[0049] The partial-dialog models outperform the full model as well asthe marginal for respective dialog turn datasets with the highestaccuracy on the first dialog turn at 94% accuracy. The full model alsoachieves 85% on the complete dataset but provides poorer results whenthe dataset is decomposed by dialog turn. It is understood thatincreases in the amount of data can boost the partial models for dialogturn over the marginal models. TABLE 3 Classification accuracy of themarginal, full, and partial-dialog models. Turn 1 Turn 2 Turn > 2Complete Model (5792) (3741) (2652) (12185) Marginal 0.6979 0.68670.5199 0.6516 Full 0.7029 0.699  0.3872 0.8514 Turn 1 0.9404 n/a n/a n/aTurn 2 n/a 0.8083 n/a n/a Turn > 2 n/a n/a 0.7164 n/a

[0050] Beyond probing the probabilistic relationships among dialogbehavior and success, a goal of the present invention has been toharness the probabilistic models in dynamic decisions about the costsand benefits of shifting a caller to a human operator. Having access toprobabilities of the eventual success of a session with an automatedsystem, as a function of the observed state of a dialog, provides acontrol surface enabling such decision-making.

[0051] Qualitatively, probabilistic models with the ability to providepredictions about outcomes provide an immediate way for administratorsof automated call routing systems to specify preferences regarding thetransfer of callers to a human operator. Such preferences can berepresented as a tolerated threshold on failure as a function of thecurrent expected time that callers will have to wait for a humanoperator, given the current load on operators. The probabilistic modelscan also be employed in call center design. Staffing decisions can bemade with the overall system model, constructed by taking intoconsideration the probabilistic performance of an automated system toroute calls successfully, preferences about wait time, characterizationof caller volumes, and the time required for addressing callers in aqueue waiting for an operator.

[0052] Systems and methods can be provided for handling the overallchallenge of optimizing a call routing system design, based on aqueue-theoretic formulation. In this invention, a decision-theoreticanalysis is provided that minimizes the expected wait time for a caller,given observations about the nature and progress of dialog with anautomated system.

[0053] Assuming that the utility for a user, u(n,m,w), associated withthe process of call routing, is a function of the number of dialog stepstaken so far with the automated system, n, the total additional expectednumber of steps that will be taken for the current call with anautomated routing system, m, and the wait time, w, associated with atransfer to a human operator should a transfer occur. It is noted that,in the general case, not only should the total time be considered, butthe nature of the interaction steps. As an example, people may beextremely frustrated with the poor recognitions of the system, even ifthe overall outcome is accurate. Beyond the number of turns and waittimes, the utility of an interaction for a caller may be influenced byother factors. For example, callers may have a negative emotionalreaction to working with an automated system versus a human operator.Such factors can be folded into a cost-benefit analysis of routingactions under uncertainty, considering the number and nature of eachstep in a dialog.

[0054]FIG. 7 is a diagram 700 illustrating prior distribution of theexpected number of dialog steps conditioned on the success of theautomated call routing system. To continue the analysis from thediscussion relating to FIG. 6, it can be assumed that the utility of aninteraction is captured by the time cost of the interaction. Theanalysis can be generalized with a conversion of steps to an effectivetotal time of an interaction, where frustration is captured by increasesin the effective total time of specific steps. Thus, in one class ofsolutions, an assumption can be made that u(n,m,w) is captured by ameasure of effective time, t(n,m,w)=t(n)+t(m)+w.

[0055] The expected number of additional steps can be computed at eachpoint in a dialog for the case where there is eventual success with theuse of the automated system, and the case where the automated routingfails for any reason (e.g., the user presses the “O” key to access ahuman operator) and the user is immediately routed to a human operator.FIG. 7 displays the expected number of steps for success at 0 steps—theoutset of an interaction.

[0056] The present invention employs p(xfer|E,ξ) to refer to theprobability that the interaction will eventually fail at some point andthe user will be immediately routed to a human operator at that point.At each point in an interaction, pre-compute and make available,p(xfer|E,ξ) and p (success|E,ξ)=1−p(xfer|E,ξ). Also pre-compute theprobability distributions, p(m|E,xfer,ξ)—the probability distributionover the additional number of steps given an eventual failure,p(m|E,success,ξ)—the probability distribution over the number ofadditional steps given a successful interaction with the automatedsystem, and the expected number of steps for each of these conditions,labeled <m> and <m′>, respectively.

[0057] The expected additional wait time with continuing the automatedinteraction, t^(a) at each point in a dialog under uncertainty infailure is as follows:

t ^(a) =p(xfer|E,ξ) (t(<m>)+w)+(1−p(xfer|E,ξ)) t(<m′>)  Equation 1

[0058] The wait time associated with an immediate transfer into thequeue for interacting with a human operator is w. This time varies andthe current value can be measured or estimated at any moment bymeasuring the average recent wait times, or by checking the queue ofcallers waiting for an operator. At the current position in a dialog,the expected wait time associated with continuing the automatedinteraction versus making an immediate automated transfer to the waitingqueue for a human operator can be compared. That is, continue todetermine if t^(a)>w. If the expected wait time is greater forcontinuing to engage the user with the automated system, then transferthe user to the queue for a human operator.

[0059] Beyond taking a myopic analysis, the present invention canperform different amounts of look ahead, and invoke Equation 1 tocompute the expected wait times at points further downstream in thedialog, folding in a consideration of the uncertainty that the user willtake different paths conditioned on the current path, and willsuccessfully reach each of the downstream points. Employing thedecision-theoretic measure promises to minimize the total wait time forusers.

[0060] Also, the cost of handling a call with a human operator, C, foran organization and take an organizational perspective on decisionmaking can be considered. Rather than just modeling the cost to a userin terms of time and frustration, the following, considers anorganization's assignment of the utility as a function the cost of thehandling of a call by a human operator, in addition to the organizationsassessment of the cost associated with the user's time and frustration,

Utility of call handling=p(xfer|E,ξ) u(t(n)+t(<m>)+w),C)+(1−p(xfer|E,ξ)) u(t(n)+t(<m′>).  Equation 2

[0061] In this aspect, consider an organization's assessment of autility, u(t, C), capturing the overall cost as a function of the waittime that a caller experiences and the monetary cost of handling a call.Note that the cost may not be constant; evidence about a user'sselections and responses may be used to update a probabilitydistribution over the cost p(C|E,ξ), and the expected costs can besubstituted in Equation 2.

[0062] As another extension, consider also the cost of a total failureof the communication, based on folding in the likelihood that a userwill simply hang up in frustration at each point in the interaction, asa function of the path they are on and the wait times in a queue. Withthis extension, the overall organization's consideration of the expectedcost of handling a call is,

Utility of call handling=1−p(fail|E,ξ) (p(xfer|E,ξ) u(t(n)+t(<m>)+w, C,success)+(1−p(xfer|E,ξ)(u(t(n)+t(<m′>),0, success)+p(fail|E,ξ)u(u(t(n)+t(<m>),0, fail)  Equation 3

[0063] In this aspect, consider the organization's assessment of autility, u(t, C, outcome), of the cost as a function of the wait timethat a caller experiences, the monetary cost of handling a call, andwhether the call eventually is completed or fails with a disconnection.The loss of a caller is denoted as p(fail|E,ξ). Such a utility model canbe expanded to include more detailed modeling about whether a caller islost to an organization forever or until another attempt is made.

[0064]FIG. 8 illustrates a methodology for call routing and decisionmaking in accordance the present invention. While, for purposes ofsimplicity of explanation, the methodology is shown and described as aseries of acts, it is to be understood and appreciated that the presentinvention is not limited by the order of acts, as some acts may, inaccordance with the present invention, occur in different orders and/orconcurrently with other acts from that shown and described herein. Forexample, those skilled in the art will understand and appreciate that amethodology could alternatively be represented as a series ofinterrelated states or events, such as in a state diagram. Moreover, notall illustrated acts may be required to implement a methodology inaccordance with the present invention.

[0065] Proceeding to 802, the present invention considers the variousinteractions and performance of a real-world speech-recognition based onan automated call routing system. As noted above, this can include ananalysis of the various system components and operator interactionsaffecting one or more variables of system performance. At 804, data isgathered to determine various performance aspects of the system. Thiscan include call answering statistics, routing statistics, success orfailure statistics based on various factors such as the amount of time acaller has to wait before being directed into a queue to speak with ahuman operator, for example. At 808, one or more probabilistic modelscan be constructed from the data that is employed to provideprobabilities of different paths through the call routing system,including such information as whether an interaction with the automatedsystem will be successful. At 812, a decision-theoretic analysis of thevalue of switching to a human operator at different points in a user'sinteraction with the automated routing system is also provided.

[0066] With reference to FIG. 9, an exemplary environment 910 forimplementing various aspects of the invention includes a computer 912.The computer 912 includes a processing unit 914, a system memory 916,and a system bus 918. The system bus 918 couples system componentsincluding, but not limited to, the system memory 916 to the processingunit 914. The processing unit 914 can be any of various availableprocessors. Dual microprocessors and other multiprocessor architecturesalso can be employed as the processing unit 914.

[0067] The system bus 918 can be any of several types of busstructure(s) including the memory bus or memory controller, a peripheralbus or external bus, and/or a local bus using any variety of availablebus architectures including, but not limited to, 11-bit bus, IndustrialStandard Architecture (ISA), Micro-Channel Architecture (MSA), ExtendedISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect (PCI), Universal Serial Bus (USB),Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), and Small Computer SystemsInterface (SCSI).

[0068] The system memory 916 includes volatile memory 920 andnonvolatile memory 922. The basic input/output system (BIOS), containingthe basic routines to transfer information between elements within thecomputer 912, such as during start-up, is stored in nonvolatile memory922. By way of illustration, and not limitation, nonvolatile memory 922can include read only memory (ROM), programmable ROM (PROM),electrically programmable ROM (EPROM), electrically erasable ROM(EEPROM), or flash memory. Volatile memory 920 includes random accessmemory (RAM), which acts as external cache memory. By way ofillustration and not limitation, RAM is available in many forms such assynchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM),double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SynchlinkDRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0069] Computer 912 also includes removable/non-removable,volatile/non-volatile computer storage media. FIG. 9 illustrates, forexample a disk storage 924. Disk storage 924 includes, but is notlimited to, devices like a magnetic disk drive, floppy disk drive, tapedrive, Jaz drive, Zip drive, LS-100 drive, flash memory card, or memorystick. In addition, disk storage 924 can include storage mediaseparately or in combination with other storage media including, but notlimited to, an optical disk drive such as a compact disk ROM device(CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RWDrive) or a digital versatile disk ROM drive (DVD-ROM). To facilitateconnection of the disk storage devices 924 to the system bus 918, aremovable or non-removable interface is typically used such as interface926.

[0070] It is to be appreciated that FIG. 9 describes software that actsas an intermediary between users and the basic computer resourcesdescribed in suitable operating environment 910. Such software includesan operating system 928. Operating system 928, which can be stored ondisk storage 924, acts to control and allocate resources of the computersystem 912. System applications 930 take advantage of the management ofresources by operating system 928 through program modules 932 andprogram data 934 stored either in system memory 916 or on disk storage924. It is to be appreciated that the present invention can beimplemented with various operating systems or combinations of operatingsystems.

[0071] A user enters commands or information into the computer 912through input device(s) 936. Input devices 936 include, but are notlimited to, a pointing device such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner,TV tuner card, digital camera, digital video camera, web camera, and thelike. These and other input devices connect to the processing unit 914through the system bus 918 via interface port(s) 938. Interface port(s)938 include, for example, a serial port, a parallel port, a game port,and a universal serial bus (USB). Output device(s) 940 use some of thesame type of ports as input device(s) 936. Thus, for example, a USB portmay be used to provide input to computer 912, and to output informationfrom computer 912 to an output device 940. Output adapter 942 isprovided to illustrate that there are some output devices 940 likemonitors, speakers, and printers, among other output devices 940, thatrequire special adapters. The output adapters 942 include, by way ofillustration and not limitation, video and sound cards that provide ameans of connection between the output device 940 and the system bus918. It should be noted that other devices and/or systems of devicesprovide both input and output capabilities such as remote computer(s)944.

[0072] Computer 912 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer(s)944. The remote computer(s) 944 can be a personal computer, a server, arouter, a network PC, a workstation, a microprocessor based appliance, apeer device or other common network node and the like, and typicallyincludes many or all of the elements described relative to computer 912.For purposes of brevity, only a memory storage device 946 is illustratedwith remote computer(s) 944. Remote computer(s) 944 is logicallyconnected to computer 912 through a network interface 948 and thenphysically connected via communication connection 950. Network interface948 encompasses communication networks such as local-area networks (LAN)and wide-area networks (WAN). LAN technologies include Fiber DistributedData Interface (FDDI), Copper Distributed Data Interface (CDDI),Ethernet/IEEE 1102.3, Token Ring/IEEE 1102.5 and the like. WANtechnologies include, but are not limited to, point-to-point links,circuit switching networks like Integrated Services Digital Networks(ISDN) and variations thereon, packet switching networks, and DigitalSubscriber Lines (DSL).

[0073] Communication connection(s) 950 refers to the hardware/softwareemployed to connect the network interface 948 to the bus 918. Whilecommunication connection 950 is shown for illustrative clarity insidecomputer 912, it can also be external to computer 912. Thehardware/software necessary for connection to the network interface 948includes, for exemplary purposes only, internal and externaltechnologies such as, modems including regular telephone grade modems,cable modems and DSL modems, ISDN adapters, and Ethernet cards.

[0074]FIG. 10 is a schematic block diagram of a sample-computingenvironment 1000 with which the present invention can interact. Thesystem 1000 includes one or more client(s) 1010. The client(s) 1010 canbe hardware and/or software (e.g., threads, processes, computingdevices). The system 1000 also includes one or more server(s) 1030. Theserver(s) 1030 can also be hardware and/or software (e.g., threads,processes, computing devices). The servers 1030 can house threads toperform transformations by employing the present invention, for example.One possible communication between a client 1010 and a server 1030 maybe in the form of a data packet adapted to be transmitted between two ormore computer processes. The system 1000 includes a communicationframework 1050 that can be employed to facilitate communications betweenthe client(s) 1010 and the server(s) 1030. The client(s) 1010 areoperably connected to one or more client data store(s) 1060 that can beemployed to store information local to the client(s) 1010. Similarly,the server(s) 1030 are operably connected to one or more server datastore(s) 1040 that can be employed to store information local to theservers 1030.

[0075] What has been described above includes examples of the presentinvention. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe present invention, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the presentinvention are possible. Accordingly, the present invention is intendedto embrace all such alterations, modifications and variations that fallwithin the spirit and scope of the appended claims. Furthermore, to theextent that the term “includes” is used in either the detaileddescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprising” as “comprising” is interpretedwhen employed as a transitional word in a claim.

What is claimed is:
 1. An automated call routing system, comprising: anautomated call routing component to route incoming calls to members ofan organization and providing automated responses to one or morecallers; and a decision model associated with the automated call routingcomponent to mitigate transferring the calls to an operator.
 2. Thesystem of claim 1, further comprising a speech recognition component forcommunicating with the callers.
 3. The system of claim 1, the decisionmodel is trained from a data log that has recorded data of pastactivities and interactions with the call routing component.
 4. Thesystem of claim 3, the data log contains data relating to at least oneof a Speaker Found, a Speaker Not Found, an OperatorRequest, a HelpRequest, a Hang Up, a Maximum number of Errrors, a Not Ready indication,and an Undefined category.
 5. The system of claim 1, the decision modelprocesses one or more dialog features including at least one of systemand user actions, session summary features, n-best recognitionsfeatures, and generalized temporal features.
 6. The system of claim 5,the n-best recognitions features are derived from a speech recognizer,and the generalized temporal features are included to cover trendsbetween one or more n-best lists.
 7. The system of claim 1, the decisionmodel employs a probability tree determining a likelihood of successgiven a sequence of system actions.
 8. The system of claim 7, thedecision model determining p(SpeakFound|E), wherein observationalevidence E refers to system actions taken, by counting a number oflogged cases along an action sequence that resulted in success over atotal number of cases along the sequence, wherein p is a probability. 9.The system of claim 1, decision model employs a dependency network thatprocesses one or more categories of dialog features as input variables.10. The system of claim 9, the decision model processes at least one ofa sequence of system actions, a count or number of alternates in ann-best recognitions list, a number of times a user attempted to speak aname, a largest score assigned by a call routing system, and a number ofdialog turns—defined as a question-answer pair.
 12. The system of claim1, the decision model employs a Markov Dependency network.
 13. Thesystem of claim 12, further comprising a component to increase an amountof data in order to boost a partial model for dialog turns over amarginal model.
 14. The system of claim 1, the decision model includesprobabilistic models to perform dynamic decisions about costs andbenefits of shifting a caller to a human operator.
 15. The system ofclaim 14, the probabilistic models provide predictions about outcomes toenable administrators of automated call routing systems to specifypreferences regarding the transfer of callers to a human operator. 16.The system of claim 15, the preferences are represented as a toleratedthreshold on failure as a function of a current expected time thatcallers have to wait for a human operator, given a current load onoperators. The probabilistic models can also be employed in call centerdesign.
 17. The system of claim 1, the decision model is employed tofacilitate staffing decisions by taking into consideration at least oneof probabilistic performance of an automated system to route callssuccessfully, preferences about wait time, characterization of callervolumes, and time required for addressing callers in a queue waiting foran operator.
 18. The system of claim 17, the queue is optimized based ona queue-theoretic formulation.
 19. A computer readable medium havingcomputer readable instructions stored thereon for implementing at leastone of the call routing component and the decision model of claim
 1. 20.A system that facilitates call routing, comprising: means forinteracting with a caller; means for automatically directing the callerto a user; and means for performing a decision theoretic analysis beforedirecting the caller to a user, the decision-theoretic includes a costbenefit analysis weighing the benefits of transferring the caller to anoperator.
 21. A method for automatically routing calls, comprising:determining a utility model for employment with a call routing system;training the utility model from a log of past system call activities;and automatically directing calls to at least one of an organizationmember and an operator.
 22. The method of claim 21, the utility model isapplied to a user function, u(n,m,w), associated with a process of callrouting, the user function is a function of a number of automated dialogsteps taken, n, a total expected number of steps that will be taken withan automated routing system, m, and a wait time, w, for transferring toa human operator.
 23. The method of claim 22, further comprisingprocessing user frustrations.
 24. The method of claim 22, furthercomprising processing negative emotional reactions to working with anautomated system versus a human operator.
 25. The method of claim 22,further comprising performing a cost-benefit analysis of routing actionsunder uncertainty, considering a number and nature of at least one stepin a dialog.
 26. The method of claim 21, further comprising determininga utility of an interaction in accordance with a time cost of aninteraction.
 27. The method of claim 26, further comprising generalizinga conversion of steps to an effective total time of an interaction,wherein frustration is captured by increases in an effective total timeof specific steps.
 28. The method of claim 26, further comprising apre-computation that is performed to yield, p(xfer|E,ξ) andp(success|E,ξ)=1−p(xfer|E,ξ).
 29. The method of claim 26, furthercomprising a pre-computation of probability distributions, p(m|E,xfer,ξ)and p(m|E,success,ξ) and an expected number of steps for conditions,labeled <m> and <m′>, respectively.
 30. The method of claim 21, furthercomprising determining an expected total wait time with continuing anautomated interaction, t^(a) at respective points in a dialog underuncertainty in failure as: t^(a) =p(xfer|E,ξ) (t(<m>)+w)+(1−p(xfer|E,ξ))(t(<m′>)wherein a wait time associated with a courteous immediatetransfer into a queue for interacting with a human operator is w. 31.The method of claim 30, further comprising determining a utility of callhandling as follows: Utility of call handling=p(xfer|E,ξ)u(t(n)+t(<m>)+w), C)+(1−p(xfer|E,ξ)) u(t(n)+t(<m′>)
 32. The method ofclaim 31, further comprising determining the utility of call handling asfollows: Utility of call handling=1−p(fail|E,ξ) (p(xfer|E,ξ)u(t(n)+t(<m>)+w, C, success)+(1−p(xfer|E,ξ)(u(t(n)+t(<m′>),0,success)+p(fail|E,ξ) u(u(t(n)+t(<m>),0, fail).
 33. The method of claim32, where cost of handling the call with a human operator C depends onneeds or goals of the caller, and is inferred from evidence.
 34. Themethod of claim 32, further comprising determining expected costs viainference of a probability distribution over Cost given evidencegathered so far, p(C|E,ξ).
 35. The method of claim 21, furthercomprising providing online sensing of current wait times for callsbeing transferred to a human operator.
 36. The method of claim 21,further comprising at least one of: creating an end-to-end system thatcontinues to at least one of log, monitor, and build models; andautomatically setting parameters, generating reports, and generatingtraces, for validation and auditing of actions.
 37. The method of claim21 supporting an application including at least one of touch-tonerouting and speech recognition.